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Acquirer/Supplier  Mismatch 
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EIA-632 


In  what  areas  does  the 
Program  office  spend 
its  technical  time? 


Acquisition 
Request  | _ ^ 


f  System 
Products 
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PMO  System  Engineering  Activities 
(Notional) 

Review  of  Contractor  Materials  (CDRLs,  IDE) 

Participation  on  IPTs 
Preparation  of  PMRs 
Risk  Management  Activities 

Probing  Contractor  Activities  for  Award  Fees  Determination 

Facilitate  Technical  Reviews  (Gov’t  only  and  Contractor) 

Plan  for  Subsequent  Years 

Manage  Government  Furnished  Property 

Manage  Functional  Baseline 

Plan  for  Spiral  Development 

Participation  in  Councils,  Boards,  and  Working  Groups 
Manage  Stakeholder  Involvement 


©  2004  by  Carnegie  Mellon  University 


page 


Carnegie  Mellon 

Software  Engineering  Institute 


OSD  System  Engineering  Focus 


Date 

Document 

Key  Points 

February  20, 
2004 

Policy  for 
Systems 
Engineering  in 
DOD 

•  Develop  a  SEP  that  describes  overall  technical 
approach,  including  processes,  resources, 
metrics,  and  applicable  performance  incentives. 

•  Detail  timing,  conduct,  and  success  criteria  of 
Tech  Reviews 

•  Director,  Defense  Systems  -  review  program 
SEPs  (where  AT&L  is  the  MDA)  as  part  of 
preparation  for  DAB  reviews. 

March  20, 
2004 

Implementing 

Systems 

Engineering 

Plans  in  DOD  - 

Interim 

Guidance 

•  Address  the  integration  of  the  technical  aspects 
of  the  program  with  the  overall  program 
planning,  SE  activities,  and  execution  tracking 
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Details  of  March  20  Guidance 

Processes  to  be  applied,  how  they  will  be  implemented  and 
tailored,  how  they  will  support  the  technical  &  programmatic 
products  required  of  each  phase. 

Technical  baseline  approach:  how  developed,  managed,  and 
used  to  control  requirements,  design,  integration,  VER,  and 
VAL.  Discuss  metrics  (TPM)  for  the  technical  effort  and  how 
they  will  be  used  to  measure  progress. 

Timing,  conduct,  success  criteria,  and  expected  products  of 
technical  reviews.  How  they  will  be  used  to  assess  technical 
maturity,  assess  technical  risk,  and  support  program  decisions. 
Updates  to  include  results  of  completed  technical  reviews. 

How  SE  activities  will  be  integrated  within  and  coordinated 
across  IPTs;  how  IPTs  will  be  organized;  what  SE  tools  they  will 
employ;  resources,  staffing,  management  metrics,  and 
integration  mechanisms;  how  SE  activities  are  integrated  in  the 
program’s  overall  integrated  schedules. 
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FY03  NDAA  Section  804  (Dec  02) 

Services/departments  shall  establish  programs  to  improve  the 
<software>  acquisition  process  ...  120  days  after  enactment 


Program  Requirements 

•  Documented  process  for  planning,  requirements  development  and  management, 
project  management  and  oversight,  and  risk  management 

•  Metrics  for  performance  measurement  and  continual  process  improvement 

•  A  process  to  ensure  adherence  to  established  process  and  requirements  related  to 
the  acquisition  of  software 

ASD(C3I)  and  USD(AT&L): 

•  Prescribe  uniform  guidance  for  implementation  across  DoD 

•  Assist  services/departments  by: 

-  Ensuring  source  selection  criteria  include  past  performance  and  the  maturity  of 
the  software  products  offered  by  potential  sources 

-  Serving  as  a  clearinghouse  for  best  practices  in  software  development  and 
acquisition  in  both  the  public  and  private  sectors 
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Purpose  of  Multi-mission  Maritime 
Aircraft  (MM A)  Program 


currently  prodded  by  the 

P-3  aircraft 
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availability  at 
the  right  cost 
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Transformation 


The  P-3  aircraft  provides  the  USN 
with  blue  water  and  littoral 
Undersea  Warfare  (USW) 
capabilities,  and  performs  armed 
intelligence,  surveillance  and 
reconnaissance  functions 


Innovative 
logistic  solutions 
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FY02-04:  Component  Advanced  Development 
•  Multiple  contracts  awarded  for  MMA  system 

-  Defined  MMA  system  architecture 

-  Validated  operational  requirements 
document  (ORD) 


FY04-13:  System  Development  and  Denipnstratipn 
•  Single  contract  awarded  for  MMA  system 

-  Design,  development  and  test  MMA  system 

-  Ground,  flight,  live  fire  test  articles 


MMA  SDD  contract  awarded  to  Boeing 
for  the  737  MMA  on  14  June  2004 
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MMA  Document  Hierarchy 
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MMA  SEP  Outline 

Introduction 

Reference  Documents 

Systems  Engineering  Process  Plan  (SEPP) 

Technical  Team  Organization 

Technical  Planning  and  Control 

Technical  Reviews 

Program  Reviews 

Spiral  Development/Technology  Transition 
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Ex.:  Systems  Engineering  Process 

Inputs 

Requirements  Analysis 
Functional  Analysis  /  Allocation 
Synthesis 
System  Analysis 
Verification  and  Validation 
Outputs 
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Ex.:  Use  of  Annotated  Outline 

PERFORMANCE  MEASUREMENT  AND  ANALYSIS 
Earned  Value  Management 

Discuss  the  EVM  contractual  requirements  we’ve  placed  on  the  vendor. 
Explain  that  a  cost  account  manager  (CAM)  counterpart  matrix  will  be 
established  at  the  IBR. 

Describe  the  technical  approach  to  determining  the  vendor  award  fee. 

Describe  how  the  government  team  will  monitor  progress  against  their  IMP  / 
IMS  and  make  decisions  based  on  status  (control).  Refer  to  the  Decision 
Analysis  and  Resolution  process  and  Technical  Management  Processes 
sections  of  Appendix  A  as  appropriate. 

Technical  Performance  Measures 

Identify  the  TPM  philosophy  for  MMA  and  identify  the  candidate  TPMs.  Use 
the  data  from  VSEMP  4.4 

Technical  Metrics 

Identify  and  expand  upon  the  goal  of  having  technical  metrics  at  the  cost 
account  level  to  augment  the  earned  value  data  coming  from  the  contractor 
EVMS.  Identify  the  types  of  technical  metrics  we  plan  to  use.  Specifically 
discuss  SW  metrics.  Grab  the  SW  metrics  chart  from  the  CAD  Software 
Development  Plan  CDRL.  For  metrics  related  to  execution  of  the  Government 
team  processes  and  IMP  /  IMS,  refer  to  the  Measurement  and  Analysis 
process  section  of  Appendix  A. 
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SEP  Process  Definition  Outline 

Conceptually  use  process  definitions  as  SOWs  for  future 
sub-tier  plans 

Use  interview  technique  to  identify  "To  Be"  process  state 

Use  a  specific  outline  for  the  process  definitions 

Create  diagrams  that  illustrate  the  relationships  between 
the  processes 

Explicit  identification  of  generic  characteristics  (measures, 
configuration  management  items,  reports,  training) 

Use  of  the  CMMI-AM  as  a  set  of  practices  that  represent 
the  PMO 
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MMA  Process  Mapping  to  CMMI-AM 


MMA  SEP  Processes 

Technical  Planning,  Monitoring  and  Control 

Solicitation  and  Contract  Monitoring 

Risk  Management 

Integrated  Teaming 

Requirements  Development  and  Management 

Integrated  Testing 

Measurement  and  Analysis 

Configuration  Management 

Decision  Analysis  and  Resolution 

Training 

Product  and  Process  Quality  Assurance 
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Generic  Process  Outline 

Introduction 

Process  Description  (with  Context  Diagram) 
Activities 

Technical  Baseline  and  Programmatic  Products 

Decisions 

Communications 

Configuration  and  Data  Management 

Metrics 

Training 
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Ex.:  Integrated  Teaming  -  Context 
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Ex.:  Support  Process  -  Context 
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Contract 
Monitoring 


Contractor 


/  Sub  \ 

/  Sub  \ 

\  #1  J 

\  #2  ) 

Training 


Status 

Re  porls.. 
Procedures 
Status 


Reports-  Status 
Procedures 


S' - N.  X  RFPs,  EVMS 

f  Sub  \  X  "'"v.  ^ 

l  #4 

Configuration 

\V_X7  sub  Ay - S?anus - 

Management 

/  l  #n  J\ 

V  y  L  CDRL&'^X^' 

Stalus 


Briefings 
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Decision  Analysis 
and  Resolution 


Configuration 

Management 


Technical 
Planning, 
Monitoring  and 
Control 


Requirements 
Development  and 
Management 


Integrated 

Teaming 


Process  and 
Product  Quality 
Assurance 
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Ex.  Integrated  Testing  -  Activities 


Task/ Activity 

Type 

Responsibility 

Sub-process? 

Develop  MMA  Test  Plans 

Event 

Team  Leads 

Yes 

Conduct  MMA  Test  Plan 
Readiness  Reviews 

Event 

Team  Leads 

Yes 

Attend  Testing  IPT  Meetings 

No 

Prepare  Facilities  Plan 

Event 

Team  Leads 

Yes 

Develop  TEMP 

Testing  Lead 

No 

Evaluate  CDRLs 

Team  Leads 

Yes 

Attend  Milestone  Reviews 

Event 

APMSE 

No 

Conduct  Product  Evaluations 

Yes 

Establish  and  maintain  the 
testing  schedule 

Team  Leads 

No 
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Ex.:  Solicitation  and  Contract 
Monitoring  -  Metrics 


Measurement 

Source 

Frequency 

Analysis  Method 

Government  Quality 

Solicitation  and 

Source  Selection 
Preparation  and 
Execution 

As  Occurring 

Survey  Bidders  and 
Source  Selection 

Team  Members 

Contractor  Quality 

CDRLs  and  other 
Work  Products 

As  Occurring 

Comparison  to  DID, 
Audit  Work  Products 

Government 

Timeliness 

Adherence  to 

Contract  Schedule, 
CDRL  Reviews 

As  Occurring 

Compare 

delivery/review  dates 
to  due  dates 

Contractor  Timeliness 

Adherence  to 

Contract  Schedule, 
CDRL  Deliveries 

As  Occurring 

Compare  delivery 
dates  to  due  dates 

Customer  (Govt.) 
Satisfaction 

PMRs 

Quarterly 

Satisfaction  Ratings 

Contractor  Satisfaction 

PMRs 

Quarterly 

Satisfaction  Ratings 
(?) 

Number  of  Bidders’ 
questions  to  RFP 

Contract  Bids 

As  Occurring 

Comparison  to  RFP 
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Ex.:  Technical  Planning,  Monitoring 
and  Control  -  Communications 


Name 

Frequency 

Type 

IMP 

Continuous 

MS  Excel 

IMS 

Continuous 

MS  Project  (or  Sigma) 

Corrective  action  list 

Weekly 

MS  Excel 

Systems  Engineering  Coordination 

Weekly 

Meeting 

MMA  Core  Team  Coordination 

Weekly 

Meeting 

MMA  Leadership  Coordination 

Weekly 

Meeting 

Task  descriptions 

A/R 

e-mail 

Team  (and  Sub-Team)  Coordination 

Weekly 

Meeting 

IPT  Coordination 

Weekly 

Meeting 

Status  information  put  into  IDE 

Weekly 

MS  Word 

Relevant  stakeholders  input  and  review 

Continuous 

MS  Word  or  email 

Critical  path  in  IMS 

Weekly 

MS  Project 
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Ex.:  Integrated  Testing  - 
Configuration  Items 


Configuration  Item 

CM/DM 

File  Type 

Expected  Update? 

Test  Plan  CDRLs 

CM 

MS  Word 

Based  on  government 
and  supplier 
comments  and 
reviews 

Testing  IPT  meeting 
minutes 

DM 

MS  Word 

After  each  meeting 

Review  meeting  minutes 

CM 

MS  Word 

After  each  review 

Test  Reports 

CM/DM 

MS  Word 
MS  Excel 

After  completion  of 
testing  effort 

Coding  Standards 

CM 

MS  Word 

Based  on  government 
and  supplier 
comments  and 
reviews 
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New  Guidance  from  OSD 

Systems  Engineering  Application  to  Life  Cycle  Phases 

•  System  Capabilities,  Requirements  and  Design  Considerations 

-  Capabilities  to  be  Achieved 

-  Key  Performance  Parameters 

-  Certification  Requirements 

-  Design  Considerations 

•  SE  Organizational  Integration 

-  Organization  of  IPTs 

-  Organizational  Responsibilities 

-  Integration  of  SE  into  Program  IPTs 

-  Technical  Staffing  and  Hiring  Plan 

•  Systems  Engineering  Process 

-  Process  Selection 

-  Process  Improvement 

-  Tools  and  Resources 

-  Approach  for  T rades 
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New  Guidance  from  OSD  -  2 

Systems  Engineering  Application  to  Life  Cycle  Phases 

•  Technical  Management  and  Control 

-  Technical  Baseline  Management  and  Control  (Strategy  and 
Approach) 

-  Technical  Review  Plan  (Strategy  and  Approach) 

•  Integration  with  Other  Program  Management  Control  Efforts 

-  Acquisition  Strategy 

-  Risk  Management 

-  Integrated  Master  Plan 

-  Earned  Value  Management 

-  Contract  Management 
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Agenda 

Process  Improvement  in  the  Program  Office 

Program  Office  System  Engineering  Activities 

MMA  Program  Context 

Program  Office  documentation  hierarchy 

Program  Office  System  Engineering  Plans 

Collaboration  Mechanisms 

How  Did  We  Integrate  Processes  with  the  SEP? 

Recent  OSD  Guidance 

Lessons  Learned 
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Lessons  Learned 

The  SEP  activity  was  new  ground,  not  much  legacy  to  draw  on...  led  to 
prototyping  and  reevaluation  of  end  state,  and  took  longer  than  we 
wanted 

The  Program  Office  recognized  the  need  for  improvement,  and  worked 
with  us  shoulder  to  shoulder  to  develop  the  SEP...  a  different  situation 
would  have  made  this  task  very  challenging 

Ideally,  the  SEP  should  be  an  evolving  document  from  an  earlier 
program  life-cycle...  OSD  guidance  points  future  programs  to  create  this 
document  early  in  the  life-cycle  and  evolve  it  as  they  proceed  from 
milestone  to  milestone 

Throughout  the  process  OSD  guidance  was  evolving...  not  an  optimal 
condition 

Be  clear  about  the  difference  between  Verification  and  Product  and 
Process  Quality  Assurance 

SEP  Prep  Guide  V  0.90  Released  18  Oct  04  by  OSD  will  help  in 
evolution  of  document  to  include  initial  release  for  future  programs 
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Recent  Updates 

Process  Improvement  Plan  has  been  developed  that 
supports  the  SEP 

•  Compelling  reasons  for  process  improvement 

•  Roles  and  Responsibilities 

•  Strategy,  Activities,  Resources,  and  Schedule 

-  Implementation  Schedule 

-  Process  Action  Plan  Skeleton 

•  MMA  Program  Context  Information 

•  Plan  Measures 

•  Plan  Risks 

•  Plan  Outputs 

•  Plan  Communications 
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Plan  Highlights 

Plan  scoped  for  five  (5)  years  of  implementation  and 
improvement  activity 

Identified  responsibility  for  three  levels  of  organization: 

•  SE  Process  Steering  group 

•  SE  Process  group 

•  Technical  Working  Group 
Approximately  three  (3)  process  per  year 

Focused  workshops  are  integral  to  the  process  action  plans 


©  2004  by  Carnegie  Mellon  University 


page  40 


